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Abstract 



The cost of error is high in the institutional research environment in terms of 
time, resources, and credibility. Therefore, institutional research professionals 
strive to produce work which is timely, error-free, accurate, and informative. A 
theme common to both total quality management and business process design is 
to replace inspection or error checking activities with improved work processes. 
This paper presents a framework for understanding how errors arise in work and 
suggests principles for creating error-sensitive work processes. To illustrate these 
principles, four redesigned projects will be described and demonstrated, 
including a survey, spreadsheet, simulation, and policy analysis. 
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Error-Proofing Work Products and Processes in Institutional 

Research Offices 



INTRODUCTION 

Institutional research professionals are familiar with the mandate to produce error-free work 
under tight deadlines with few resources. The pressures for error-free work and the 
consequences of error are well known. A misplaced decimal point, ill-conceived analysis, or 
unrecognized concept can wreak havoc with the reputation and credibility of professionals. 
However, despite the importance of avoiding errors, there are few practical guidelines available 
to professionals on how to do so. 

A theme common to total quality management and business process design is to reduce 
inspection or error checking activities by better product design and improved work processes. 
As Stratton (1991) has pointed out, finding, fixing, and fighting errors is very wasteful compared 
to preventing errors. There are clear payoffs in both quality and efficiency by dedicating more 
time to product design than to product checking. Better work processes can also lower waste, 
restarts, and rework. In a recent study of errors committed by institutional researchers, the 
authors found that nearly 60% of all reported errors were not discovered until well beyond the 
time or stage of work at which the errors had been produced. The average lag for detecting 
errors was nearly 25 days. Clearly, successful re-design of work products and processes in 
institutional research offices could close the gap between error production and error discovery. 
This paper proposes some principles for designing work products and processes less likely to 
result in errors, and shows how these principles were successfully applied on four different 
projects. 

UNDERSTANDING ERRORS 

It is almost as difficult to define "error" as it is to define "quality." Definitions of quality have 
focused on two main ideas: (1) product performance or "fitness for use" (Juran, 1989, p. 15); and 
(2) customer satisfaction or "meeting customers' needs and reasonable expectations" (Berry, 
1991, p. 3). Quality processes are also important in evaluating a product. According to Chafee 
and Sherr (1992, p. v), "a quality process means that all the steps within the organization's 
functioning, from beginning to end, work effectively toward the desired goals, with each step 
adding value." 
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If quality can be defined in terms of product fitness, product process, and client satisfaction, 
what then are errors? In this paper, an error is a product which fails to meet professional 
standards, has unacceptable process limitations, or is unacceptable to a client for one of four 
reasons: (1) inadequate conceptualization (understanding of the project goals or research 
question was flawed); (2) poor design or method (an ineffective approach was taken to 
answering the research or project question); (3) flawed process (the strategies, tactics, or 
procedures chosen to implement the design for the product were flawed); or (4) poor 
presentation (the presentation or packaging of the product was ineffective). 

In stating that a product may be in error simply because it is unacceptable, we are implicitly 
recognizing that errors are not always objectively identifiable. Errors are not always as easy to 
identify as a mistake in addition or a transposed number. Senders and Moray (1991, p. 19) 
claim that human error "does not refer to something observable, in the same sense as 
decision-making does." Errors are always inferred from the results of processes. Errors can be 
matters of judgment, based upon the meanings attributed to data or reports or other products. 
A report may be accurate, for instance, but irrelevant to the need at hand. Such subtleties are 
not unusual especially in reference to policy analysis. A given policy analysis may fail as a 
product in many subtle ways. The analysis may be correct but not politically operable. The 
proposed policy may speak correctly - to the wrong issue. The policy solution may work - but 
not in the environment where it has been proposed. 

HOW ERRORS ENTER INTO WORK PROCESSES 

Understanding how errors enter into work processes is the first step to making significant 
improvements. There are four interactive components to any work process: (1) the 
environment in which the work process operates; (2) the agents implementing the process; (3) 
heuristics or the written and unwritten rules governing how the process operates; and (4) the 
tools which are applied to materials to produce work. Consider a study of tuition and fees as an 
example of a work process. The agents in the institutional research office are the staff trying to 
collect the tuition and fee information, and those with whom the staff interact to do so. The 
heuristics are governed by a pre-defined form for collecting the information, an unwritten 
"script" for asking the questions, and a set of procedures for checking and producing the final 
analysis of data. The tools used range from language to paper and pencil to a computer 
spreadsheet. The materials are the data received from the other offices, and the outcome is a 
report. Feedback from the environment about the report eventually flows back to the agents, 
who can then modify the process in response to the feedback. 



Chart 1 shows the relationship between each of these factors in influencing process outcomes: 



Chart 1: System View of Work Processes 



Under this model, agents work 
within an environment to define 
the rules of the game for using tools 
to produce outcomes or results. 
Feedback about the outcomes of 
the work process form the basis for 
future work. 




Errors can arise at any of the stages shown above: 

Environment . All work occurs within a particular environment. The environment is often given 
names like "campus climate/' ''work climate/' "home team advantage/' or "office politics." 
The environment circumscribes all of the internal and external fadors which can influence how 
work is done or received. Examples of internal factors include resources, work values, the time 
available to do the work, and training. Many external factors, such as politics, audit standards, or 
legal considerations, may also influence the environment or expectations for work. 

Agents . Every person has a unique perception of the world arising from different cultural, social, 
psychological, and personal experiences. In addition, each person's base of knowledge and 
contextual understanding will differ. For all of these reasons, agents may contribute to errors in 
numerous ways. Earl Babbie (1990) lists several ways in which agents can err, including 
inaccurate observation, over-generalization, selective observation, made-up information, 
illogical reasoning, ego-involvement in understanding, premature closure of inquiry, and 
mystification. 

Heuristics . Heuristics describe the way in which a process, activity or task will operate. The 
process may be rational or irrational. The "rules of the game," "standard operating 
procedures," "decision rules/' "rules of engagement/' and "office protocols" all convey some 
sense of the broad range of heuristics. The way in which the heuristics operate for a task may 
contribute to errors or reduce the usefulness of work products. For instance, estimating annual 
enrollments at an institution by multiplying IPEDS fall enrollment data by a factor of two will fail 
if an institution's pattern of enrollments does not fit a standard academic semester calendar. 

Tools . The term "tools" is used in its broadest sense to describe the means by which the work 
inputs for a project are translated into an outcome or product. The means can be a mind, a 
pencil, a calculator, or an electronic tool like a computer. Tools can produce an error primarily 
if misused or misinterpreted by the agent. Tools can also represent inadequate approaches to 
accomplishing a task. A needle and thread will not bind together a book, any more than a 
policy without support will govern behavior. 
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STRATEGIES /TECHNIQUES FOR CREATING ERROR-SENSITIVE WORK PROCESSES 

Four principles are suggested in this paper for making work processes more error-sensitive 
and error-free: (1 ) clarify environmental constraints as part of the work process and use this 
understanding in designing the work to be done; (2) empower agents to overcome 
organizational, perceptual, or knowledge base limitations in defining, managing, or interpreting 
the results of work processes; (3) design heuristics which emphasize openness of information 
and feedback loops ; and (4) use technologies which generate intraprocess feedback. 

Clarifying Environmental Constraints 

Errors arise from internal and external environmental constraints. Internal constraints have 
sometimes been called the "organizational climate for work". Tagiura and Litwin (1968) define 
organizational climate as "the relatively enduring quality of the total internal environment of an 
organization that (a) is experienced by the members, (b) influences their behavior, and (c) can 
be described in terms of the values of a particular set of characteristics (or attributes) of the 
environment" (Tagiura and Litwin, 1968, p. 27). 

Litwin and Stringer (1968) identified several key components of organizational climate 
which can impact performance: (1) responsibility (degree of delegation experienced by 
employees); (2) standards (expectations about work quality); (3) reward (recognition for good 
work versus disapproval for poor performance); (4) organizational clarity (orderliness versus 
disorderliness); and (5) team spirit (fellowship and trust within an organization). Performance 
can be stimulated (and errors minimized) in organizations which stress meaningful delegation 
to employees (Chaffee and Sherr, 1992), emphasize the importance of quality in every part of 
the organization (Berry, 1991), recognize and reward quality (Winter, 1991), show constancy of 
purpose (Deming, 1986), and encourage teamwork (Chaffee and Sherr, 1992). 

External environmental constraints help to define the "acceptability" of work products. 
These constraints may be political, legal, professional, or arise from other sources. For instance, 
a policy paper aimed at restructuring compensation practices will need to take into account 
the effect of changes on a broad range of actors, including staff (who would be directly affected 
by any change), individuals in the larger job market (who will respond to future changes), the 
general public (who may bear the costs for the changes in a public system), the legislature (who 
may have other approaches or priorities), other employers (who may resent increased 
competition if wages increase), or other institutions (which may respond in kind to policy 
changes, thereby changing competitive dynamics in the market). An analysis which focuses only 
on cost or form of change in compensation practices may result in too limited a view, and 
underestimate the reaction to a policy change - which may ultimately doom the proposed 
policy to failure. It may not be possible to take into account all external constraints impacting a 
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work product, but at least, clarifying the constraints can help avoid serious problems. 
Recognizing and assessing constraints early in the work process also allows time to handle and 
manage the expectations concerning the work more effectively. 

Empowering Agents 

Traditional hierarchical organizations fov.us most resources for professional development on 
the top of the organization, hoping for "trickle-down" effects for other staff. Gilbert and Nelson 
(1991) have urged those concerned with quality in organizations to focus instead on 
empowering employees throughout the organization. 

"The quality approach assumes the people closest to the process know what the problems are 
.... They have good ideas about how to improve things so they can do their jobs better. They 
are encouraged to discuss them and be heard by others with whom they work, and they are 
able to improve things in collaboration with others on their teams. They do not need heroic 
leaders to solve their problems. They are partners in problem solving with their customers, 
suppliers, and others in their teams who can help (Gilbert and Nelson, 1991, p. 132)." 

Gilbert and Nelson suggest that the emphasis in organizations needs to be on "followership" 
not leadership; the qualities associated with followership are partnership, motivation, technical 
competence, dependability, professional comportment, sense of humor, positive working 
relations, and willingness to speak up (Gilbert and Nelson, 1991). - 

Peter Senge carries the concept of followership further in advocating the need for learning 
organizations, or "organizations where people continually expand their capacity to create the 
results they truly desire, where new and expansive patterns of thinking are nurtured, where 
collective aspiration is set free, and where people are continually learning how to learn 
together (Senge, 1990, p. 3)." Such organizations provide the climate for empowering 
employees to create higher quality, more error-free work. Learning organizations are 
organizations which are open, or characterized by a high degree of information sharing, and 
which nurture the freedom to interpret information in novel ways. In such organizations, 
employees are empowered by access to information and lack of constraints on discussing its 
meaning for the organization. 

Many total quality management perspectives stress that most errors are due to 
management decisions about how to organize work rather than worker mistakes (Deming, 
1986; Stratton, 1991). Organizational structures or climates may also contribute heavily to the 
production of errors or of low quality work. However, human agents are very well positioned to 
discover and correct errors, irregardless of the real causes of the errors. 

Most of the errors which can be attributed to people, arise because of organizational, 
perceptual, or knowledge base limitations in defining, managing, or interpreting the results of 



work processes. Empowering staff to do better work may require additional training 
opportunities, better tools, or better access or understanding of the context for tine work being 
done. Training can address knowledge base limitations. Tools can aid employees in many ways, 
but can be especially valuable in overcoming perceptual limitations in spotting or detecting 
errors. The most effective way to help staff overcome contextual limitations is to create more 
openness in the work organization, thereby allowing staff to better understand the context in 
which work is performed. Effective policy analysis, for instance, can not occur in a vacuums- 
policy studies are hampered by lack of access to decision makers and other stakeholders in the 
external environment. 

Re-designing Heuristics 

Heuristics govern a wide range of behaviors related to how organizations, and the people in 
them, function. Heuristics are the rules, explicit or implicit, which set expectations for behavior 
in given situations or contexts. The rules are given many names - laws in science, norms in 
sociology, mores in anthropology, etiquette in social relations. Senge (1991, p. 174) has called 
these types of rules mental models, or "deeply held internal images of how the world works." 
The mental models are important in that they not only constrain or limit ideas, but they also 
color perceptions. Not only may individuals favoring a particular mental model not consider 
other approaches but they may also fail to perceive their existence. 

On an organizational level, heuristics can be instrumental in determining patterns of 
openness for information and exchange of ideas. Openness is one of the critical characteristics 
of a learning organization (Senge, 1991); in such organizations members are not only free, but 
are actively encouraged, to explore novel solutions to problems and to improve the ordinary 
ways of doing the work of the organization. Hierarchy can have the opposite effect. 
Hierarchical relations constrain information access and limit the ability to re-think work based 
on one's position within the organization. At the project or task level, heuristics biised on 
hierarchy limit feedback on work. For example, it is important for staff who produce reports or 
analyses to have a clear understanding of the use of those products in order to make the work 
accurate and relevant. Yet, in a hierarchical organization, staff may not be invited to meetings 
where their work products are used or discussed. In such circumstances, staff may receive 
indirect feedback from those in attendance, but lack direct insights into how their work met 
the needs of their client. 

It is important to challenge and re-define the heuristics governing how work is done in a 
given setting if those heuristics do not foster openness and a rich environment for feedback 
about the work being done. As Peter Senge (1991) has noted, it may be necessary to challenge 
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the prevailing mental model goveming a task or activity before any improvements can be made 
in the task. 

Using Technologies to Emphasize Feedback 

Feedback is especially critical to the improvement of work processes - whether in industry 
or institutional research. Feedback which occurs after the product is already developed is useful 
for future improvements, but feedback obtained during the work process, or intraprocess 
feedback, can help to avoid errors or improve quality. 

Technology can effectively provide intraprocess feedback. The instant replay is an example 
of feedback obtained during a sports game which is used to promote quality of decision making. 
Referees can use the replay to make better decisions during the game rather than waiting for 
appeals or rulings after the game. Another example is a smoke detector. Smoke detectors 
provide feedback early in the fire process with the aim of alerting people to the fire before the 
situation becomes dangerous. The yellow light at a street crossing is another example of a 
technology designed to aid decision-making in the middle of a process (crossing an intersection) 
rather than at the beginning (green light) or the end (red light). 

In institutional research, it is desirable to detect errors while the work is in the office rather 
than in the client's office, or in a publication. Frequently an error is not discovered until the 
intensive error-checking just prior to the release of a study or report. At that point, more effort 
is required to fix the error than if the error had been identified soon after it was made. 

RE-DESIGNING WORK PROCESSES 

The principles of clarifying environmental constraints, empowering, re-designing heuristics, 
and using tools which emphasize feedback can be used in institutional research tasks to 
improve work processes. Four examples of project re-design are given in this paper: (1 ) a survey 
on student charges; (2) a comparative analysis of faculty salaries; (3) an enrollment planning 
process; and (4) re-definition of a comparator group. 

Example 1: Student Charges Survey Re-design 

The University System of New Hampshire (USNH) is composed of four institutions and a 
branch campus. The main institutions, the University of New Hampshire, Keene State College, 
and Plymouth State College, have long operated as "privatized" publics, or institutions whose 
state funding is modest and whose primary source of revenue is tuition. As such, it is important 
to regularly monitor tuition, fee, and other student charges for USNH institutions compared 
with those of other institutions with whom there is competition for enrollments. To this end, 
the Office of Policy Analysis conducts a survey of the tuition and fees of roughly thirty institutions 
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twice a year. The information fronn the survey is used by the Board of Trustees in setting and in 
approving tuition and fee charges. Published sources can not be used for this purpose because 
such sources are generally retrospective and do not provide information for charges for the 
coming academic year. 

In the past the survey was done using paper survey forms to record the results of telephone 
conversations with staff from other institutions. Table 1 contains a copy of the form which had 
been used. Fax transmissions were used both to obtain and confirm data from the participants 
in the sur\'ey. The data gathering form was re-designed several times and the survey questions 
refined, yet survey results often did not track with data from other sources. Analysis of outcomes 
indicated the errors resulted from an inability to translate questions into the "jargon" of 
particular institutions or for the surveyer to remember details about the way each institution 
handled and reported student charges. Sometimes several offices within an institution had to 
be contacted to obtain the needed information (for instance, the registrar's office for tuition 
and the housing office for dormitory fees). It was not always possible to survey the same 
individual in a given office from year to year. Finally, it often took several iterations of telephone 
calls to obtain, clarify, troubleshoot, and correct information. One reason for the need to revisit 
information initially obtained during a sur^/ey interview was that it was difficult to identify an 
error until the information obtained had been summarized and compared to responses from 
prior years. 

A new approach was developed which i'uv.v.sed on empowering the surveyer in several 
ways: 1) to request and clarify information using the institution's terms and "jargon"; 2) to have 
information available while asking questions which would describe any special circumstances or 
nuances which the surveyer should address; 3) to get immediate feedback showing how the 
information received during the survey interview compared to prior historical data from the 
institution; 4) to automatically calculate and annualize the data obtained so that information 
could be reflected to the person being interviewed for purposes of clarification or confirmation; 
and 5) to record clarifying information received during the interview for future use. The goal of 
the re-design of the survey process was to enable the surveyer to obtain as much accurate 
information as possible from each telephone call made and to reduce the number of call backs 
needed for correcting or clarifying information. 

The re-design was accomplished by using a computer to enter and get feedback on 
information as it was received on the phone. An ORACLE data structure was created to accept 
the data as they were obtained on the telephone. SQL Forms were used to provide contact 
information, create data entry screens, initiate programs for processing and presenting the 
information, and generate feedback to the surveyor on the information received. Conlextual 
information needed to frame questions, enable immediate data entry, show the 
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TABLE !• Initial Student Charges Survey Form 



FY95 FUIL'-TIME UNDERGRADUATE STUDENT CHARGES 
UNHAND COMPARATORS 



NAME OF INSTITUTION: 
CONTACT PERSONS: 



TELEPHONE NO.:_ 
FALL & SPRING 



TOTALSTUDENT CHARGES: 

Resident: 

Non-Resident: 



DATE OF SURVEY:. 



STAFF 



What was the percent increase in student charges over the previous 
year? 



I — I ESTIMATED AS OF_ 
I I FINAL AS OF 



Are there any unusual increases or additional student charges compared to the previous year?_ 
How many semesters/quarters/other make up your academic year: 



Data for collection represents fiilltime, undergraduate resident and non-resident student charges for one academic year. 
CATEGORY 



FULL-TIME Undergraduate RESIDENT 
TUITION (tuition only; Do Not include 
mandatory fees) 

How many hours per semester are 
you using to calculate the tuition 
charge? 



Hours per 
semester « 



$ Per semester/quarter/ 
other 



X 



Total for Category per 
Academic Year 



FULL-TIME Undergraduate NON-RESI- 
DENT tuition (tuition only; Do Not include 
mandatory fees) 

Standard Double-Occupancy Dorm Room: 

Meal Plan: Standard 19 meals per week. 
Or your plan that comes closest 

TOTAL MANDATORY FEES: (fees thut all 
student<i must pay in order to attend classes) IX2l 
include tuition. 



X 
X 
X 



Mandatory Student Fee Breakdown (Plcaseshwsubtotalofmmdatorysndmtfccsbyancgcfy. Addcatcgort^ ifnccd:d) 



STUDENT ACTIVITIES: 
STUDENT SERVICES: 
STUDENT HEALTH/ACCID. 
STUDENT REC. 
STUDENT UNION 



Per Semester 


Per Year 























EDUC. TECH. 
REGISTRATION 
INTERCOL. ATH. 
COMPREHENSIVE FEE 
TRANSPORTATION FEE 
STUDENT GOVT FEE 

Other categories specific 
to your institution 



Per Semester 


Per Year 







































University System of New Hampshire Office of Policy Analysfe 
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reasonableness of data received in light of historical data, and provide error-trapping with range 
edits was included on the screens for data entry. Table 2 contains an example of the main data 
entry screen from the program. The new approach is being pilot tested to see if call-backs are 
reduced and the survey completed more efficiently. 



TABLE 2. Screen From Interactive Survey Form 

STUDENT CHARGES- 



UNnriD Institution Name 

[UNIVERSITY OF NEW HAMPSHIREAl\IN CAMPUT 



^ AY Code ENTRY DATE 

nn EH] CZ3 lo1-M^Y■94l 



RES: 
NRES: 



Amount 
I 1835 



Mult 



Annual 



HR 



PerHR 



152.9 



ROOM: I 1163 | 
BOARD: I R'ify J 



I 11990 
I 2326 I 
I 171? I 



] QO [ 



499.6 



PRIOR YEAR- 



UNHID 
jl 83044) 

RES: 
NRES: 
ROOM: 
BOARD: 



AYR Code 



-2L 



Amount Mult 



5995 I 
I 1163 I 



I R'if, 



COMP ENTRY DATE 

1 ^ \ I 01 ■M\Y-94 

Annual 
1 3670 I 



MOST RECENT TERM- 
AY Code ENTRY DATE 
95 I m |01-MAY>94| 



HR 



PerHR 



12 



I 11990 I 



152.9 



12 



2326 



M717 



T 



-TUITlOrvJ- 



$ Dif % Inc $ Dif % inc 
RES:!!)! in NRES: fTi 



$Dif 

[o] 



-CURRENT YEAR— 
.ROOM BOARt 



% Inc 

El 



$ Dif % Inc 



-FEES- 



I OldTO T NewToT $Dif % Inc 



Example 2: Salary Study Process Re-design 

Every year, the Office of Policy Analysis produces a book summarizing the salary market 
position of faculty within the University System of New Hampshire by discipline. To do so, 
survey data are obtained from the Oklahoma State University study of faculty salaries by 
discipline. The study provided by Oklahoma State University is a summary of the responses of 
institutions designated as comparators for the University of New Hampshire (one database) and 
for the State Colleges (a second database). Not all comparator institutions respond to the 
Oklahoma State University survey. Special surveys requesting similar data are sent to non- 
responding comparator institutions. In the past, processing of the data from the surveys 
occurred in several steps: 1) the Oklahoma State University and special survey data are 
combined into a single database on a VAX platform, by CIP code; 2) the data are analyzed to 
extract only those CIP codes for which there are programs at the University of New Hampshire 
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or the State Colleges; 3) the CIP codes are analyzed and re-grouped where necessary to assure 
adequate numbers of cases in each grouping to be studied; 4) salary data are sorted and 
organized by CIP code for the University of New Hampshire and State College data and the 
relevant comparator databases; 5) two files are extracted with the salary data (one for the 
University of New Hampshire comparator data and one for the State College comparator data) 
and transferred into a Macintosh microcomputer environment; 6) calculations are made on 
salary deficiency and position to market using an Excel spreadsheet on the Macintosh; 7) data 
from the Excel spreadsheet are imported into Pagemaker and are used to produce the final 
report (which uses multiple graphs and tables per page in the report to summarize the 
comparisons.) Table 3 contains a sample page from the salary book report. 

The complex process described had shortcomings in two main areas. The first was the study 
data from Oklahoma State University could not easily be checked for accuracy until combined 
and then processed into CIP groupings. Any population errors or discrepancies persisted well 
into the processing of data. The second, and more serious problem, was that it was difficult to 
spot errors or inconsistencies concerning the results of the salary analyses until the last two 
stages of work, when calculations and visual summaries were available. Any redo of work which 
had progressed to the last stage meant considerable effort to fix the problem and then re- 
import the new data streams to produce graphics in Pagemaker. Not atypically, the error 
capture process was most effective in the very last stages of work, rather than in the early work 
processes. The third shortcoming was that using multiple platforms to process and package the 
data resulted in a complex process for correcting errors when identified. 

An early attempt to re-design the work process addressed the issue of multiple platforms. 
The process was re-designed so that all database creation, processing, and report preparation 
was done on a single platform, the VAX. A pilot project was undertaken to produce the salary 
study for the 1991 year. While the salary book was successfully produced using a combination 
of 1032 and Interleaf software, the pilot wa?, not repeated. The process did not improve the 
ability to trap errors earlier in the work process, and data transfer into the final production 
product, Interleaf, required considerable programming effort. Interleaf proved to be a highly 
specialized product which was difficult to use without extensive training. Production of the 
report under this approach would have required too much external computing support and 
allowed too little control of the production process by the Policy Analysis staff. 

Re-design of this work process incorporated two principles. The first principle was to use 
technology to front-load error-trapping earlier into the processing of data from Oklahoma State 
University and from external surveys. For this purpose, one program was developed on the VAX 
to compare data received from the external sources with past year data. Total counts were 
analyzed, allowing for a quick assessment of the reasonableness of the study population. 
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TABLE 3: Sample Page from USNH Salary Study Report (Mock Data) 



UNH 

Discipline: 



Mathematics 



CIP Code: 



270000 



All Academic Year Faculty 
Full Profcs5or 
Associate Professor 
Assistant Professor 

AH Fiscal Year Facully+ 
Full Professor 
Associate Professor 
Assistant Professor 



All Academic Year Faculty 
Full Profcoor 
AsscK iaie Professor 
Assistant Professor 

AH Fiscal Year Faculty^ 



AV<) 5 Sal.irv Avor<itios 



UNH Number of 
Average Faculty 



Average of 
Comparators 



Number of 
Faculty 



% Above^Below 
Comparator Average 



$70,973 12 
$53,483 6 
$43,896 5 



$0 
$0 
$0 



$70,031 
$52,322 
$44,618 



$70,031 
$52,322 
$44,618 



193 

95 

58 

193 

95 

58 



AY93 Discipline 
Salary Base * 



$851,675 
$320,895 
$219,481 

$0 



AY93 Salary Base Using Percent Change Needed 
Comparator Averages ^* For Salary Base Catch-Up 



$840,375 
$313,910 
$223,088 

$0 



AY93 Totals 

$1 ,392,05 1 $1 ,377,393 

•Salary ba« e«imal<»d as number UNH faculty times salary avcfage fof each rank. 

••Salary hoe estimated as n; mbci of UNH faculty times compjralor salary average for each rank. 

MY93 cjhmaic of fijcal faculty bast'd on .857 of actual FY93 salary. 



-1.33% 
-2.17% 
1.64% 

0.00% 
-1.05% 



1.34% 
2.22% 
-1.62% 

0.00% 
0.00% 
0.00% 



Discipline Salary Summary 

■H AY93 Oncipltne SaUry Bate 
AY9J Salary Baic ining 
Comparator Av«fag« 



$U92X)5t $1377393 
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This process identified errors due to the composition of external data, which had proved a 
problem in the past. Table 4 contains an example printout from this program. Another 
program was written to compare counts by CIP code and CIP groupings for the current and past 
year in order to quickly identify potential problems with the external data. A final program 
produced the actual deficiency calculations on the VAX, rather than within the Macintosh 
Excel software. All of this programming enabled efficient assessment of the accuracy of the data 
well before any graphics were produced. 

The second principle was to improve the heuristics governing the data handling process. 
Current practice was to create a speedy mockup of the study report so that data checking could 
begin. Deadline pressure was on the office to produce the report quickly in support of 
collective bargaining efforts. The report is routinely audited by an external agency, such as 
Coopers and Lybrand, so accuracy was also critical. The new heuristics called for extensive error 
and reasonableness checks of the data prior to any mockup of the report pages. The Excel 
spreadsheet became the main tool for this part of the process. A new master spreadsheet was 
designed so that information was categorized and highlighted by visual clues. Data summary 
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BEST COPY AVAILABLE 



16 



columns were added to the spreadsheet which calculated percent totals and percent increases. 
The summary columns gave immediate feedback on keyboard inputting accuracy and were 
highly visible flags for out-of-range data. Additionally, color text was used to highlight CIP codes 
that required a calculation outside of the normal parameters. Embedded spreadsheet notes 
were added (a feature of Microsoft Excel which allows for voice and written notes to be assigned 
to specific cells). The notes were used to annotate key formulaes, explain important exceptions 
to data handling rules, and point out potential pitfalls in certain areas of the analysis. Cells with 
^otes are automatically marked with a red dot signal to alert the staff member to potential 
problems and provide specific information. 



TABLE 4: Excerpts from Ou!;.ut of Population Assessment Program 



CIP 

Major 

Group 


Faculty 
Rank 


FY93 
Totals 


FY94 
Totals 


Difference 
FY94-FY93 
Totals 


%incr. 
FY93-FY94 


45 


00170 


151 


194 


43 


28.5% 




00171 


107 


115 


8 


7.5% 




001 72 


87 


96 


9 


1 0.3% 




00173 


6 


3 


-3 


-50.0% 


Total 




351 


408 


57 


16.2% 


50 


00170 


122 


124 


2 


1.6% 




00171 


83 


76 


-7 


-8.4% 




00172 


80 


87 


7 


8.8% 




00173 


15 


11 


-4 


-26.7% 


Total 




300 


298 


-2 


.0% 


51 


00170 


10 


16 


6 


60.0% 




00171 


36 


30 


-6 


-16.7% 




00172 


33 


33 


0 


.0% 




00173 


14 


8 


-6 


-42.9% 


Total 




93 


87 


-6 


-6.5% 


52 


00170 


70 


109 


39 


55.7% 




00171 


88 


101 


13 


14.8% 




00172 


67 


76 


9 


1 3.4% 




00173 


1 


1 


1 


.0% 


Total 




226 


287 


61 


27.0% 


ALL CIPS 












Grand 












Total 












1 0 to 52 




2507 


2589 


82 


3.3% 
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This effort to re-design the work process on the faculty salary study has been successful. 
Data errors are identified earlier in the process and corrected before extensive investment of 
staff time occurs. More improvements will be made. For instance, it may be possible to 
eliminate the step of exporting data into Pagemaker for final report production by making full 
use of the graphic capability of Excel. 

Example 3: Enrollment Modeling Re-design 

Enrollment planning has a strong flavor of "demographic determinism" at most institutions. This 
doesn't arise out of any conviction that such forecasts are very accurate. Carol Frances 
(1989) has explained at length why such forecasts are often misused and later found to be 
inaccurate in higher education planning. Reliance on demographic forecasts arises more from 
a lack of proven alternatives than from dogmatism about the method. A departure from the 
common heuristics underlying enrollment planning, however, has its risks. New methods which 
are not widely familiar or accepted can be difficult to "sell" as creditable. 

Similar to other institutions, the University System of New Hampshire maintains and 
publishes a set of enrollment projections based upon college-going rates of traditional age high 
school graduates. In 1994, these projections indicated that significant increases in demand 
from New Hampshire students could be expected from the mid 1990s through the early years 
of the next century. Recognition of this challenge spurred a major enrollment planning effort in 
the system. 

No one involved in the planning process was entirely satisfied that simple linear 
demographic projections could show enough detail about future demand patterns to be useful. 
There was scepticism about the accuracy of the projections and a desire to incorporate, or at 
least recognize, some of the other factors which would ultimately influence enrollments. 
Capacity, the economy in New Hampshire and New England, competition from other 
institutions, and the demand from nontraditional students are a few examples of such factors. 
Consequently, one of the first steps taken in the planning process was to adopt an ambitious 
new approach to refining the understanding of potential demand for New Hampshire public 
higher education. This break in heuristics culminated in the hiring of a consulting firm, 
Cambridge Decision Dynamics, to work with staff from the Office of Policy Analysis in using a 
new technology for support of planning, a system dynamics model of enrollments. 

Moving to a new approach presented three challenges. First, the creditability of the model 
would be challenged since the technique was new and would not-mirror previous work. 
Second, a new knowledge base was needed on the part of those using the model if it was to 
serve as a viable policy simulator. Third, the model itself was sufficiently complex that new 
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communication and teaching tools were needed if policy makers were to become fully 
engaged in its use. 

The process design for implementing the model used all of the principles for designing error 
free work processes. The technology change was the key to changing the mental model or 
heuristics for enrollment planning. System dynamics modeling is a relatively new planning 
technique for higher education, although it has been used in other fields for about three 
decades (Senge, 1991). The model developed is a microworld, or computer replica of the 
many external and internal factors affecting the University System of New Hampshire and its 
institutions. This microworld contains quantitative as well as qualitative, or "soft" variables 
reflecting the best judgments of professionals in institutional operational areas such as 
admissions, student services, and financial aid. The scope of the model is broad, with major 
subsystems covering institutional policy and data (finance, space, admissions, academic policy, 
financial aid, personnel, students), market share (college-going rates in local and regional 
markets), enrollments (admissions patterns, academic aptitude, numbers of students, transfer 
patterns, retention patterns, exit and graduation patterns), demographics, K-12 sector, 
economy, and state policy. The model was fully arrayed by both race and residency of students. 
A diagram showing the many factors incorporated in the system dynamics model by Cambridge 
Decision Dynamics is included as Chart 2. 

The change in heuristics was supported by meetings and demonstrations at each campus. 
The new technology was explained and participation was encouraged in a broad data gathering 
effort to support the creation of the system dynamic model. An additional consultant was hired 
specifically to support campus use of the model and other planning tools in developing 
strategies to deal with the foreseeable enrollment demand. Feedback from the model was an 
integral part of the planning process. The ultimate goal of the project was to put into place a 
model at each campus to be used as a planning tool both for the enrollment planning project 
as well as for future planning. 

The structure of the model helped to clarify environmental constraints on solutions to the 
enrollment problem. Using prior methods, policy makers were left to speculate on the 
potential impact of both external and policy choice factors on enrollment demand. A 
recession, change in federal financial aid policy, or migration trends could all affect enrollments, 
but there was no clear method for identifying and quantifying the impact. Under the new 
technology, the effect of such factors on enrollments could be estimated with greater precision. 

The model serves to empower decision makers by creating an environment for policy 
simulation. As was the case with external factors, decision makers could only guess at the likely 
impact of internal policy choices such as changes in the amount of institutional aid provided to 
students or changes in admissions standards under the previous enrollment planning method. 
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Chart 2. Overview of Model System Dynamic Model Subsystems 




Source: Cambridge Decision Dynamics 



With the new technology, the effect of internal policy changes in areas such as admissions, 
financial aid, retention, transfer, and academic standards can be modeled. The simulation 
environment helps uncover consequences of policy choices which were either unrecognized 
or counter-intuitive. 

The results of this process re-design will take some time to assess. The initial model 
development for the planning process (a systemwide model) is being followed by creation and 
equilibration of models for each campus (spring-summer, 1 995). The real test of success of the 
re-design will be v/hether or not campus executives use the models as part of their internal 
planning processes. 

Example 4: Comparator Group Re-design 

In the late 1980's, the University System of New Hampshire Board of Trustees formally 
adopted comparator groups for both the University of New Hampshire and the State Colleges. 
The comparator groups were identified using consultants to guide the process. The process was 
highly participative, with involvement from the administration and academic senate of each 
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institution, as well as staff from the Chancellor's Office. However, by the early 1 990's, it was 
clear that the composition of the groups was very controversial, and had become a subject for 
discussion in the collective bargaining sessions with faculty at two of the institutions within the 
system. The main use of the comparator groups was the establishment and tracking of salary 
benchmarks, so there was a renewed interest on the part of faculty in the process. While it was 
certainly debatable whether or not the groups could be considered somehow "in error", it was 
clear that the current groups were not of positive benefit in promoting shared goals or 
understandings on salary deficiencies. Eventually, leadership in the Chancellor's Office and the 
institutions agreed that it was pointless to take a defensive stance with regard to the current 
comparator list and that it was desirable to try a new approach. The approach was to initiate a 
collaborative effort to redefine comparators. The change in heuristics ^s underway and seems to 
be leading to a successful redefinition of the comparator group for the State Colleges. 

Analysis of the conflict over the established comparator group revealed several common 
complaints. First, despite the clear documentation that showed extensive participation by 
faculty in selecting comparators, faculty denied that adequate consultation had occurred. 
Second, faculty did not believe that the group had enough representation from New England 
and/or the Northeast. Third, faculty believed that some factors which had not been considered 
in the initial process were important (for instance research productivity) and that other factors 
were given undue importance (size of institution). Finally, faculty desired input into the actual 
uses of the information from comparators such as in setting salary benchmark goals. Faculty 
complained they had never really understood how the comparator group would be used and 
thus had offered earlier feedback based on different perceived goals for the group. 

The underlying heuristic changes made as part of the re-design process are important to 
clarify. The basic change was a move from seeing faculty as participants in the process to seeing 
faculty as collaborators in the process. Under the participation model, faculty could offer advice 
and counsel, but for faculty to have any decision-making role in selecting a group against which 
their performance and salaries would be judge;! was seen as inappropriate. In seeing faculty as 
collaborators, administrators were recognizing that shared ownership of salary goals was more 
important than maintaining a hierarchical stance with regard to faculty. 

The emphasis on re-designing the process to select comparators was on creating the 
opportunity for full engagement of faculty representatives in the process. To this end, a study 
group was formed with representatives of the Chancellor's Office, Keene State College, and 
Plymouth State College to select new comparators for the State Colleges. The group was 
composed of faculty from both institutions, and administrators representing finance, planning, 
human resources, and policy analysis. To foster trust, a partnership was formed between the 
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Director of Policy Analysis and a faculty representative. All data and information supplied to the 
group are jointly created and endorsed by these two individuals. 

Another emphasis in the new process is to clarify expectations concerning comparisons in 
order to create more viable reports for use in the bargaining process. ?ssues about the 
processing and use of any salary information gathered from comparators are discussed openly in 
group meetings. 

The study group has proceeded on its task with the following steps: 1) define the criteria for 
selecting comparators; 2) weight the criteria; 3) collect information from candidate institutions; 
4) identify the best matches to the criteria; and 5) make recommendations on a filial list of 
comparators. At the time of this writing, the study group has recently issued a report 
recommending composition of a new comparator group. This report will be sent forward to 
the Chancellor and eventually to the Board of Trustees Capital and Strategic Planning 
Committee. 

The ultimate test of whether the re-design of process has succeeded will be whether or not 
the faculty support the salary goals and related benchmarks developed in the future by the 
Board of Trustees. It is likely that success in collaborating on the new comparator set may open 
the door to further collaboration on other issues and goals in the future. Another success 
measure will be whether or not the overall methodology used to study salaries will be accepted 
by faculty, given that they will now have had input and discussion of the processing and use of 
the data. 

CONCLUSIONS 

Institutional research professionals are adept at producing v^ork under time pressures with 
relatively light staff support. Errors, when they occur, are understandable, yet have important 
consequences for institutional research offices. Sensitivity to how errors enter into work 
processes can help professionals re-think and re-design how work is done within their office. 
The principles of empowering agents, re-designing heuristic rules, using tools to create process 
feedback during work, and clarifying environmental influences can be used to minimize work 
errors and enhance work quality. Institutional research professionals should look beyond 
common conceptions of errors to strive to produce work which better meets the needs of both 
operational units and policy makers. In this paper, four practical examples of institutional 
research work process improvements were described. The creation of error-sensitive work 
processes helps institutional researchers reduce errors, redos, restarts, and other inefficient 
activities in our resource-constrained environment. 
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